xHell

We are given an Excel spreadsheet with around 80 rows and five columns. The values in the A column are parameters (we will see that these are instructions later), and columns B to E are computed using a large formula, except in the first row, where they are user-provided parameters.

The formula is (example from D14, line breaks and indentation for clarity):

=IF(MID($A14,4,1)=TEXT(COLUMN()-1,"0"),
IF(MID($A14,1,1)="1",
    MOD(SUM(
        INDIRECT(ADDRESS(ROW()-1,1+MID($A14,2,1))),
        INDIRECT(ADDRESS(ROW()-1,1+MID($A14,3,1)))),256),
IF(MID($A14,1,1)="2",
    MOD(
        SUM(INDIRECT(ADDRESS(ROW()-1,1+MID($A14,2,1))),
        -INDIRECT(ADDRESS(ROW()-1,1+MID($A14,3,1)))),256),
IF(MID($A14,1,1)="3",
    MOD(PRODUCT(
        INDIRECT(ADDRESS(ROW()-1,1+MID($A14,2,1))),
        INDIRECT(ADDRESS(ROW()-1,1+MID($A14,3,1)))),256),
IF(MID($A14,1,1)="4",
    MOD(MOD(
        INDIRECT(ADDRESS(ROW()-1,1+MID($A14,2,1))),
        INDIRECT(ADDRESS(ROW()-1,1+MID($A14,3,1)))),256),
IF(MID($A14,1,1)="5",
    MOD(BITAND(
        INDIRECT(ADDRESS(ROW()-1,1+MID($A14,2,1))),
        INDIRECT(ADDRESS(ROW()-1,1+MID($A14,3,1)))),256),
IF(MID($A14,1,1)="6",
    MOD(BITOR(
        INDIRECT(ADDRESS(ROW()-1,1+MID($A14,2,1))),
        INDIRECT(ADDRESS(ROW()-1,1+MID($A14,3,1)))),256),
IF(MID($A14,1,1)="7",
    IF(INDIRECT(ADDRESS(ROW()-1,1+MID($A14,2,1)))
    =INDIRECT(ADDRESS(ROW()-1,1+MID($A14,3,1))),
        1,
        0),
"X"))))))),
D13)

We can see that this reads an instruction from the A column (a four-digit number), splits it into digits and uses them to compute the values in that row. Let’s name the digits, in order, dest, x, y and op.

We can treat the spreadsheet as an execution trace for a program which has four registers b, c, d and e, and instructions given in the A column. The instructions are decoded as:

The cell which tests if our parameters are accepted checks for three conditions. For initial values of the registers, we have b - c == 46 and e - d == 119. Further, the final value of e must be 1. The first two checks let us compute c and d from b and e, so we can try all possible values for these two registers (from 0 to 255, so this doesn’t take long) to find one that satisfies the final check and get the flag.