Token Optimization (Tokenomics)

Compressing prompts for LLM inputs

Natural language is redundant, so you can save tokens by deleting the right words and the LLM will still understand it.

LLMLingua gets 20x compression on GSM8K with only 1.5 points of accuracy lost.

Core idea: Rank tokens by perplexity, which acts as a proxy for information content. Keep the high perplexity tokens, and remove the low ones.

The two ends of that ranking are:

High perplexity tokens: hard to predict, surprising tokens. These are the ones worth keeping.
Low perplexity tokens: obvious tokens. These are the ones that get removed.

A small model is enough to do this ranking, because if a small model can predict which token comes next, so can a big model.

Output after that step is not human readable. Eg: "Sam bought a dozen boxes, each with 30 highlighter pens." gets turned into "bought boxes x0 oflters".

Simply summarizing a prompt will not give you such reduction in tokens.

★ A prompt essentially may have 3 parts: instructions, questions, and demonstrations. You cannot equally compress all 3, because they do not carry the same density of information:

Instructions and questions are the least compressible.
Demonstrations are the most compressible.

To figure out how much of each to keep, have a budget controller, where $\tau$ = keep rate:

\tau_{ins} = 0.85 \longrightarrow \text{keep 85\% of the instruction}

\tau_{que} = 0.9 \longrightarrow \text{keep 90\% of the question}

A compression ratio of 20x means:

20 = \frac{1}{\tau} \implies \tau = 0.05

You give the system the overall $\tau$ . This fixes the global allowance of $\tau \cdot L$ . Instruction and questions take their first cut, and this is a fixed amount. After this, if token limit remains, it goes to demonstrations.

Q: Consider a GSM8K full-shot prompt: 2366 tokens, 8 demonstrations. You give instructions 15 tokens, questions 60 tokens, leaving 2290 for demos. Say you want to compress 5x, therefore $\tau = 0.2$ .

\text{Global allowance } (\tau \cdot L) = 0.2 \times 2366 = 473 \text{ tokens}

\text{Instruction} = 0.85 \times 15 = 13

\text{Question} = 0.9 \times 60 = 54

\text{Tokens left for demos} = 473 - 13 - 54 = 406 \text{ tokens}

This leaves the demonstrations with a keep rate of:

\tau_{demos} = \frac{406}{2290} = 0.177

∴ Instruction and questions were kept at 85 and 90%, but demos got squeezed to 17.7%.

Formula for $\tau_{demos}$ : $\tau_{demos} = \frac{\tau \cdot L - (\tau_{ins} \cdot L_{ins} + \tau_{que} \cdot L_{que})}{L_{demos}}$

This asymmetry in compression is what matters.

★ Iterative token-level compression works in three steps:

Compute perplexity of every token (with original context).
Rank perplexities.
Delete tokens based on perplexity rank.

The problem is that once step 3 deletes tokens, the original context assumed in step 1 is now missing, therefore the perplexities are now different again.

Fix: Segment the prompt, take the first segment (maybe the first 100 tokens), compress it, then compress the next segment with the context of the compressed and uncompressed segments together.