any errors/timeouts that impact a job should be included in the CDM log summary
We recently migrated a large number of tables with a customer and noticed the counts were off on several, but they reported no errors.
We discovered they were scraping only the log summary, which showed a successful run, for example, our DiffData run showed:
`25/06/26 11:57:30 INFO JobCounter: ################################################################################################
25/06/26 11:57:30 INFO JobCounter: Final Read Record Count: 218842
25/06/26 11:57:30 INFO JobCounter: Final Mismatch Record Count: 0
25/06/26 11:57:30 INFO JobCounter: Final Corrected Mismatch Record Count: 0
25/06/26 11:57:30 INFO JobCounter: Final Missing Record Count: 43398
25/06/26 11:57:30 INFO JobCounter: Final Corrected Missing Record Count: 43398
25/06/26 11:57:30 INFO JobCounter: Final Valid Record Count: 175444
25/06/26 11:57:30 INFO JobCounter: Final Skipped Record Count: 0
25/06/26 11:57:30 INFO JobCounter: Final Error Record Count: 0
25/06/26 11:57:30 INFO JobCounter: ################################################################################################`
and our migrate job showed:
`25/06/24 02:53:00 INFO JobCounter: ################################################################################################
25/06/24 02:53:00 INFO JobCounter: Final Read Record Count: 175436
25/06/24 02:53:00 INFO JobCounter: Final Write Record Count: 175436
25/06/24 02:53:00 INFO JobCounter: Final Skipped Record Count: 0
25/06/24 02:53:00 INFO JobCounter: Final Error Record Count: 0
25/06/24 02:53:00 INFO JobCounter: ################################################################################################`
However, if you looked through the logs, you saw there were other errors/timeouts like:
`25/06/24 02:53:00 ERROR CopyJobSession: Error with PartitionRange -- ThreadID: 99 Processing min: 1763508733446635150 max: 1767198082261377060
com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT2M
at com.datastax.oss.driver.api.core.DriverTimeoutException.copy(DriverTimeoutException.java:36)
`
I believe we're tracking something different in the Error Record Count, but if we could add a line for errors that occur during the run, that would alert folks that are scripting the effort to migrate a lot of tables that they need to go look at the logs for the failed or incomplete run.
I'll upload logs from both a migrate run and a DiffData run we had trouble with
<!-- Failed to upload "diffdata.log" -->
[migrate.log](https://github.com/user-attachments/files/20930454/migrate.log)
关闭于 2025-07-08 2 条评论