Skip to navigation

Elite on the BBC Micro

Extended text tokens

The extended text token system in the enhanced versions of Elite

All versions of Elite use a clever text tokenisation system to store the game's text in an efficient manner - you can read all about it in the deep dive on "Printing text tokens". In addition to this system, the enhanced disc and 6502 Second Processor versions of Elite have two additional extended text token systems that provide a lot more text, supporting missions, disc access menus, extended system descriptions and more.

The two extended text token systems are as follows:

  • There are 256 recursive tokens in the TKN1 table that can be printed with the DETOK routine. This is the bulk of the extended token system, and contains any game text that isn't already covered by the standard text tokens or the special extended descriptions in RUTOK.
  • There are 27 special extended system descriptions in the RUTOK table that can be printed with the DETOK3 routine. These override the procedurally generated descriptions for a small group of systems, typically during the two missions when they are used to guide the player towards their mission briefings and goals (though there are some non-mission descriptions in there that provide some interesting Easter eggs for the player to find).

To print an extended token, we simply put the token number into the accumulator and call either DETOK or DETOK3. It's a lot simpler than the encoding system we have to use with TT27 for the standard tokens, though under the hood, the extended token system is just as complicated...

Types of extended token
-----------------------

Just like the standard text tokens, the tokens in the TKN1 and RUTOK tables are themselves composed of different types of token, though this complexity is hidden inside the routine that does the actual printing. This routine is known by two names, which are aliases of each other: TT26, which shares its name with the character printing routine in the cassette version, and DASC, which points to exactly the same routine. We'll talk about DASC here, as it's a slightly friendlier name.

Like the standard token system, with its control codes, two-letter tokens and recursive tokens, there are quite a few different types of extended token that DASC prints. They are:

  • Jump tokens: Instead of printing, these tokens call the corresponding routine from the jump table at JMTB. These can do anything from setting the letter case to rotating ships on screen while waiting for key presses.
  • Characters: These are standard ASCII characters, with the case determined by the extended token flags.
  • Random tokens: These are used to display the procedurally generated extended system descriptions, which use the random number generator to generate random sequences of tokens. This randomness can be controlled by seeding the random number generator before printing, which is how we ensure each system always has the same description.
  • Extended recursive tokens: These work in the same way as the recursive tokens from the standard text token system, allowing us to include tokens within tokens.
  • Extended two-letter tokens: These work in the same way as the two-letter tokens in the standard text token system, but with a larger range of two-letter sequences that extends the standard set, at the expense of dropping four of the original tokens.

As with the standard token system, the type of token is determined by the character code that is stored in memory. So, in the same way that control codes in standard text tokens are in the range 0-13, jump tokens in the extended text tokens are in the range 1-31, and random tokens are in the range 91-128. Here's a breakdown of the code ranges:

CharacterMacroProcess
1-31EJMPCall the corresponding JMTB routine
32-64ECHRPrint numbers and punctuation with TT27
65-90ECHRPrint letters A-Z in the correct case with DASC
91-128ERNDPrint an extended recursive token with DETOK, fetching the token number from the MTIN table (subtract 91 to get 0-37 then add random 0-4)
129-214ETOKPrint an extended recursive token with DETOK
215-255ETWOPrint an extended two-letter token from table TKN2 (subtract 215 to get 0-40)

As with the standard text tokens, let's make things easier to follow by referring to the four token types like this, where n is the character code:

  {n}           Jump token                n = 1 to 31
  [n?]          Random token              n = 91 to 128
  [n]           Recursive token           n = 129 to 215
  <n>           Two-letter token          n = 215 to 255

Also like the standard text tokens, the extended text tokens are stored in memory in an obfuscated manner, though this time they are EOR'd with the value of the VE configuration variable (&57) rather than the EOR 35 that is used to hide the standard tokens. To make the source code easier to read, we use various macros to assemble the tokens into memory while retaining some level of human readability of the source code. For extended tokens, the names all start with an "E", and they are:

  ECHR n          Insert ASCII character n        n = 32 to 99
  EJMP n          Insert jump token n             n = 1 to 31
  ERND n          Insert random token n           n = 91 to 128
  ETOK n          Insert recursive token n        n = 129 to 215
  ETWO 'x', 'x'   Insert two-letter token "xy"    "xy" is in the table below

Let's look at each of these types in turn, but before we do, it's worth noting that as part of the extended tooen system, it's possible to switch from extended tokens back to standard text tokens, and then back again, all within one extended token (we do this using jump tokens 5 and 6). When standard tokens are enabled, the DASC routing does the following instead:

CharacterMacroImplementation
1-31EJMPCall the corresponding JMTB routine
32-255RTOKPrint a standard text token with TT27

This behaviour is controlled by bit 7 of the print flag in DTW3: if it is clear then extended tokens are enabled, and if it set then standard tokens are enabled.

Let's now take a look at the various types of token that make up the extended text token system.

Extended recursive tokens: [n]
------------------------------

Extended recursive tokens work in the same way as standard recursive tokens, in that tokens can contain other tokens. However, the range of extended tokens that can be included in other tokens is a bit smaller than in the standard system, where you can include all but three tokens recursively. There are 256 extended tokens in the TKN1 table that the DETOK routine can print, but only tokens in the range 129 to 215 can be included in other tokens.

Apart from this, recursive tokens expand in the same way as in the standard system, and tokens can contain tokens that contain other tokens, recursing as deep as you like.

Extended two-letter tokens: <n>
-------------------------------

Also similar to the standard token system, the extended two-letter token system is based on the range of standard two-letter tokens from the table at QQ16, but with an additonal set of tokens, and four of the original tokens dropped. The full range of extended two-letter tokens is as follows:

  215     {crlf}
  216     AB
  217     OU
  218     SE
  219     IT
  220     IL
  221     ET
  222     ST
  223     ON
  224     LO
  225     NU
  226     TH
  227     NO
  228     AL
  229     LE
  230     XE
  231     GE
  232     ZA
  233     CE
  234     BI
  235     SO
  236     US
  237     ES
  238     AR
  239     MA
  240     IN
  241     DI
  242     RE
  243     A?
  244     ER
  245     AT
  246     EN
  247     BE
  248     RA
  249     LA
  250     VE
  251     TI
  252     ED
  253     OR
  254     QU
  255     AN

Tokens 215 to 227 are exclusive to the extended token system, while the standard tokens start at 228. They have token numbers that are 100 higher than the same tokens in the standard system, which is why the last four standard tokens are not available in the extended list, as they would have to have token numbers higher than 255.

The new two-letter tokens can be found in the table at TKN2, which appears directly before the standard two-letter token table at QQ16. This means we can subtract 215 from the token number to get a number in the range 0-40, which acts as an index into the TKN2/QQ16 table when doubled (as each entry takes up two bytes).

Random tokens: [n?]
-------------------

Random tokens are encoded with values in the range 91-128. When DASC is asked to print a random token in this range, it subtracts 91 from the token number to get a number in the range 0 to 37, and then it fetches the corresponding entry from the table at MTIN, adds a random number in the range 0-4 to this number, and calls DETOK to print that token.

The ERND macro, which we use to encode random tokens in the TKN1 and RUTOK tables, takes an argument between 0 and 37, which corresponds to the lookup value in MTIN.

Random tooens are used to generate the extended descriptions for each system. For example, the entry at position 13 in the MTIN table (counting from 0) is 66, so ERND 14 will expand into a random token in the range 66-70, i.e. one of "JUICE", "BRANDY", "WATER", "BREW" and "GARGLE BLASTERS".

Jump tokens: {n}
----------------

Jump tokens do exactly that - they call subroutines instead of being printed. The jump token is a very powerful token type, and implements all sorts of functionality, from drawing boxes and setting letter case, to justifying text and fetching input from the keyboard.

Jump tokens are in the range 1 to 31, though tokens 20 and 31 don't do anything. The best way to work out what each token does is to visit the relevant routine in the source. Here is a list of jump tokens, along with the subroutines that they call (the MT routines are listed in more detail below):

Jump tokenShorthand in documentationRoutine
1{all caps}MT1
2{sentence case}MT2
3{selected system name}TT27
4{commander name}TT27
5{extended tokens}MT5
6{standard tokens, sentence case}MT6
7{beep}DASC
8{tab 6}MT8
9{clear screen}MT9
10{lf}DASC
11{draw box around title}NLIN4
12{cr}DASC
13{lower case}MT13
14{justify}MT14
15{left align}MT15
16{drive number}MT16
17{system name adjective}MT17
18{random 1-8 letter word}MT18
19{single cap}MT19
20UnusedDASC
21{clear bottom of screen}CLYNS
22{display ship, wait for key press}PAUSE
23{move to row 10, white, lower case}MT23
24{wait for key press}PAUSE2
25{incoming message screen, wait 2s}BRIS
26{fetch line input from keyboard}MT26
27{mission captain's name}MT27
28{mission 1 location hint}MT28
29{move to row 6, white, lower case}MT29
30{white}WHITETEXT
31UnusedDASC

Here's a list of MT routines that implement the bulk of the jump token functionality. The number of the MT routine corresponds to the jump token that triggers that routine.

RoutineFunction
MT1Switch to ALL CAPS when printing extended tokens
MT2Switch to Sentence Case when printing extended tokens
MT5Switch to extended tokens
MT6Switch to standard tokens in Sentence Case
MT8Tab to column 6 and start a new word when printing extended tokens
MT9Clear the screen and set the current view type to 1
MT13Switch to lower case when printing extended tokens
MT14Switch to justified text when printing extended tokens
MT15Switch to left-aligned text when printing extended tokens
MT16Print the character in variable DTW7
MT17Print the selected system's adjective, e.g. Lavian for Lave
MT18Print a random 1-8 letter word in Sentence Case
MT19Capitalise the next letter
MT23Move to row 10, switch to white text, and switch to lower case when printing extended tokens
MT26Fetch a line of text from the keyboard
MT27Print the captain's name during mission briefings
MT28Print the location hint during the mission 1 briefing
MT29Move to row 6, switch to white text, and switch to lower case when printing extended tokens

The MT routines use a number of extended print flags to store the current text state. The best way to work out what each flag does is to read the relevant variable's header in the source. They are as follows:

LocationFunctionSet by
DTW1A mask for applying the lower case part of Sentence Case to extended text tokensMT1, MT2, MT13
DTW2A flag that indicates whether we are currently printing a wordCLYNS, DASC, MT8, TTX66
DTW3A flag for switching between standard and extended text tokensMT5, MT6
DTW4Flags that govern how justified extended text tokens are printedMT14, MT15, MESS
DTW5The size of the justified text buffer at BUFDASC, MESS, MT14, MT15, MT17
DTW6A flag to denote whether printing in lower case is enabled for extended text tokensMT1, MT2, MT13
DTW7Contains the character printed by MT16CATS
DTW8A mask for capitalising the next letter in an extended text tokenDASC, MT19