ITADN

any errors/timeouts that impact a job should be included in the CDM log summary

#373Closedpaayers 创建于 2025-06-26
P
paayerscommented
We recently migrated a large number of tables with a customer and noticed the counts were off on several, but they reported no errors. We discovered they were scraping only the log summary, which showed a successful run, for example, our DiffData run showed: `25/06/26 11:57:30 INFO JobCounter: ################################################################################################ 25/06/26 11:57:30 INFO JobCounter: Final Read Record Count: 218842 25/06/26 11:57:30 INFO JobCounter: Final Mismatch Record Count: 0 25/06/26 11:57:30 INFO JobCounter: Final Corrected Mismatch Record Count: 0 25/06/26 11:57:30 INFO JobCounter: Final Missing Record Count: 43398 25/06/26 11:57:30 INFO JobCounter: Final Corrected Missing Record Count: 43398 25/06/26 11:57:30 INFO JobCounter: Final Valid Record Count: 175444 25/06/26 11:57:30 INFO JobCounter: Final Skipped Record Count: 0 25/06/26 11:57:30 INFO JobCounter: Final Error Record Count: 0 25/06/26 11:57:30 INFO JobCounter: ################################################################################################` and our migrate job showed: `25/06/24 02:53:00 INFO JobCounter: ################################################################################################ 25/06/24 02:53:00 INFO JobCounter: Final Read Record Count: 175436 25/06/24 02:53:00 INFO JobCounter: Final Write Record Count: 175436 25/06/24 02:53:00 INFO JobCounter: Final Skipped Record Count: 0 25/06/24 02:53:00 INFO JobCounter: Final Error Record Count: 0 25/06/24 02:53:00 INFO JobCounter: ################################################################################################` However, if you looked through the logs, you saw there were other errors/timeouts like: `25/06/24 02:53:00 ERROR CopyJobSession: Error with PartitionRange -- ThreadID: 99 Processing min: 1763508733446635150 max: 1767198082261377060 com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT2M at com.datastax.oss.driver.api.core.DriverTimeoutException.copy(DriverTimeoutException.java:36) ` I believe we're tracking something different in the Error Record Count, but if we could add a line for errors that occur during the run, that would alert folks that are scripting the effort to migrate a lot of tables that they need to go look at the logs for the failed or incomplete run. I'll upload logs from both a migrate run and a DiffData run we had trouble with <!-- Failed to upload "diffdata.log" --> [migrate.log](https://github.com/user-attachments/files/20930454/migrate.log)
关闭于 2025-07-08 2 条评论