[WIP] Ai24 4 raise context limit #9
Conversation
|
I created this just to review it ; Mike said it's not ready yet |
There was a problem hiding this comment.
Looks good Mike! I left some notes to help out John as he reviews.
I wasn't able to trigger the nice warning message just by copy/pasting in large amounts of text though eventually I got the alert popup. Putting in large messages though it did display. Maybe it's because of the "every 8 character" rule I didn't trigger it just copy/pasting into the text area. I do like the visual style and placement of the warning message and how the conversation becomes orange in the conversation tree though I noticed some weird behavior with that (see below).
Maybe Maureen could think about the wording to use "approx. characters left in conversation context:81588 ? " is what is says now. Maybe "Warning: you are approaching the number of words this model is able to handle. Consider starting a new conversation". Would be better? Just thinking many people are not tech savvy. Maybe a different message if the initial message is the large one "This message is too large for this model".
Keep it up Mike!
components/Chat/ChatInput.tsx
Outdated
| @@ -84,9 +97,38 @@ export const ChatInput = ({ | |||
| } | |||
|
|
|||
| setContent(value); | |||
|
|
|||
| // only run the token count every 8 characters since it slows down the display of what's typed | |||
There was a problem hiding this comment.
I wonder what @Scr1ptcat thinks but consider just using the number of characters instead of calculating the tokens. Although the tokens is more accurate; given that this message is more of a warning and also that the server-side token counting may not match this "tiktok token" counting library anyway, I am not sure getting the 100% precise amount is needed.
I think as a heuristic 4 characters per token is used as you noted elsewhere.
Our GFE is quite slow and bogged down so even a little big of lagginess while people are typing would infuriate people I think.
I also think this would simplify the codebase a little bit.
components/Chat/ChatInput.tsx
Outdated
| handleChange(event); | ||
| }, [selectedConversation]); | ||
|
|
||
| //useEffect(() => { |
There was a problem hiding this comment.
Consider removing commented out code
| @@ -6,6 +6,14 @@ import tiktokenModel from '@dqbd/tiktoken/encoders/cl100k_base.json'; | |||
| import { NextApiRequest, NextApiResponse } from 'next'; | |||
| import { Tiktoken } from '@dqbd/tiktoken'; | |||
|
|
|||
| export const config = { | |||
There was a problem hiding this comment.
I'm curious about this change!
| @@ -18,7 +18,7 @@ export const OpenAIModels: Record<OpenAIModelID, OpenAIModel> = { | |||
| [OpenAIModelID.GPT_4]: { | |||
| id: OpenAIModelID.GPT_4, | |||
| name: 'GPT-4', | |||
| maxLength: 128_000*3, | |||
| maxLength: 128_000 * 4, | |||
There was a problem hiding this comment.
Good catch. Originally they were magic numbers and it appeared to be 3x the tokenLimit. Not sure why Mckay (OG author) chose 3 instead of 4x as the character upper bound.
| "version": "1.0.14", | ||
| "resolved": "https://registry.npmjs.org/@dqbd/tiktoken/-/tiktoken-1.0.14.tgz", | ||
| "integrity": "sha512-R+Z1cVYOc8ZoDls6T2YhlUYrwKyuZoRJsSK3vN7iWWjBJ1xoX7e5BhUkEh5n6cXuMWQVUTHLlSDpnyv0Ye7xxw==" | ||
| "version": "1.0.20", |
There was a problem hiding this comment.
fyi @Scr1ptcat I did see an error trying this patch but npm i updating the dependancies resolved it
| { (isPastCharacterCount || isHighCharacterCount) && ( | ||
| <span className="helpCircle" title="Once past the context limit, the conversation will no longer produce responses relevant to content before the limit"> ? </span> | ||
| )} | ||
| </div> |
There was a problem hiding this comment.
FYI here's how it appears.
I was able to get it now testing further, I just needed to type in text to trigger the once every 8 logic. Mike and I discussed using the throttling function from lodash to resolve this. |
I was getting weird errors so I simplified the logic on this page
This reverts commit 703efae.
|
I just reverted the local dev and file upload changes so this would be a clean patch/branch with this feature and then those reverts can be reverted again and tested after this patch! They are valuable changes. |
|
Looks good to me |
No description provided.