tracking: context limit issues with @-mentioned files

[abeatrix]

This is an issue spinned off from #2965 where the original goal was to display warnings for the @-mentioning massive files

Context: https://sourcegraph.slack.com/archives/C05AGQYD528/p1710331429302729

Current state

Here is the current state of @-mentioning massive files summarized by @chillatom:

1. The file size "check" that we use to display the warning is not the real token limit applied by the system. It is an approximation. This leads to the odd behavior where we trigger a warning, but you can still include the file or you can successfully copy and paste the file contents into the chat message.
2. The warning check is computed per file, which means that if file A & B are both individually under the estimated limit, we trigger no warning, even if A+B is above the limit.
3. If the file is actually too large to be included, but the user submits anyway (e.g. vscode/src/completions/logger.ts which is 860 LOC and ~6,700 tokens. It appears that we silently exclude the file from context and Cody hallucinates. 🔴
4. If the file triggers the warning, but isn't too large to be included e.g. vscode/src/local-context/symf.ts which is 641 LOC and 4,867 tokens, I see the content actually used as context and referenced correctly.

At a minimum it feels like we should do a few things

1. Tie our warning to the actual operation of the product. If we say it's too large, the file should not be able to be input
2. We should not silently exclude a file that has been explicitly @ mentioned
3. Consider the case of multiple @ mentioned files
4. Prioritize @ mention files over other fetched context (not sure if we do this today) as to avoid silently removing the file explicitly referenced

Over a longer term, I think we should explore

• Expanding the context windows, especially for some of the flagship models (I'll be writing a proposal here)
• Consider summarization or proposition extraction from the file

Design Tasks

Design Tasks:

Beta Give feedback

1. We should not silently exclude a file that has been explicitly @ mentioned - figma

Engineering Tasks

Engineering Tasks:

Beta Give feedback

1. Tie our warning to the actual operation of the product. If we say it's too large, the file should not be able to be input
2. We should not silently exclude a file that has been explicitly @ mentioned #3522
   cody +1
3. Prioritize @ mention files over other fetched context (not sure if we do this today) as to avoid silently removing the file explicitly referenced
4. Bug: Super large message in Cody is completely ignored silently with no context and an added fake preamble #3364
   bug clients/jetbrains cody +3
5. Chat: fix at-mention token size #3526
   +0
6. Chat: update @-input token background #3548
   +0
7. Chat: display excluded @-files in UI #3528
   +0
8. Chat: Disable adding large-file via @-mention #3523
   +0
9. Support range for large @-mention files #3589
   cody +1
   

Other design ideas from @toolmantim : #3439 (comment)

[taylorsperry]

Just a note that once we've aligned on a path forward and those changes have shipped, we should ping @MaedahBatool to make sure the docs are up to date. (We know users have been confused about this.)

[kalanchan]

landed in v1.12