Data ingestion fails (400 error) when text_token_length exceeds batch_token_limit

### This issue is for a: ``` - [X] bug report - [ ] feature request - [ ] documentation issue or request - [ ] regression (a behavior that used to work and stopped in a new release) ``` ### Minimal steps to reproduce Upload the Digital Project Development Manual PDF found on the Connecticut DOT website using either local ingestion or using the new cloud ingestion. This is the only PDF (out of many) that throws this error for me. - Website: [CT DOT](https://portal.ct.gov/DOT/Engineering-Applications/Digital-Project-Development-Manual?language=en_US) - Direct link to PDF: [Digital Project Development Manual](https://portal.ct.gov/dot/-/media/dot/aec/[digitalprojectdevelopmentpdf.pdf](https://portal.ct.gov/dot/-/media/dot/aec/digitalprojectdevelopmentpdf.pdf?rev=7df3b7a334f24f0eaf171cb12ef613fe)?rev=7df3b7a334f24f0eaf171cb12ef613fe) or [digitalprojectdevelopmentpdf.pdf](https://github.com/user-attachments/files/25188454/digitalprojectdevelopmentpdf.pdf) ### Any log messages given by the failure Web Api skill response contains errors: 'Web Api response contains both data and errors. Will not process Data.;Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens, however you requested 9175 tokens (9175 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}}'. ### Expected/desired behavior The document should be successfully ingested without exceeding the model token limit. ### OS Windows 11 ### azd version azd version 1.23.3 ### Mention any other details that might be useful I first encountered this issue several weeks ago while using local ingestion. I waited to see whether switching to the cloud ingestion architecture would resolve it, but the same error persists. From my investigation, I was able to trace it to the file `embeddings.py` in the def `split_text_into_batches`. I believe part of the error is in the logic around the following statement: `if batch_token_length + text_token_length >= batch_token_limit and len(batch) > 0`. At the very least, I think this is where the error is surfacing. If `text_token_length > batch_token_limit` but `len(batch) = 0` then the text gets appended to `batch` even though it's greater than the `batch_token_limit`. I'm sure this goes even deeper into the chunking logic, but I haven't found the time yet to chase it all the way down.