Class: LL::Lexer
- Inherits:
-
Object
- Object
- LL::Lexer
- Defined in:
- lib/ll/lexer.rb
Overview
Ragel lexer for LL grammar files.
Class Attribute Summary (collapse)
-
+ (Object) _ll_lexer_eof_trans
private
Returns the value of attribute _ll_lexer_eof_trans.
-
+ (Object) _ll_lexer_from_state_actions
private
Returns the value of attribute _ll_lexer_from_state_actions.
-
+ (Object) _ll_lexer_index_offsets
private
Returns the value of attribute _ll_lexer_index_offsets.
-
+ (Object) _ll_lexer_indicies
private
Returns the value of attribute _ll_lexer_indicies.
-
+ (Object) _ll_lexer_key_spans
private
Returns the value of attribute _ll_lexer_key_spans.
-
+ (Object) _ll_lexer_to_state_actions
private
Returns the value of attribute _ll_lexer_to_state_actions.
-
+ (Object) _ll_lexer_trans_actions
private
Returns the value of attribute _ll_lexer_trans_actions.
-
+ (Object) _ll_lexer_trans_keys
private
Returns the value of attribute _ll_lexer_trans_keys.
-
+ (Object) _ll_lexer_trans_targs
private
Returns the value of attribute _ll_lexer_trans_targs.
-
+ (Object) ll_lexer_en_main
Returns the value of attribute ll_lexer_en_main.
-
+ (Object) ll_lexer_en_ruby_body
Returns the value of attribute ll_lexer_en_ruby_body.
-
+ (Object) ll_lexer_error
Returns the value of attribute ll_lexer_error.
-
+ (Object) ll_lexer_first_final
Returns the value of attribute ll_lexer_first_final.
-
+ (Object) ll_lexer_start
Returns the value of attribute ll_lexer_start.
Instance Method Summary (collapse)
-
- (Object) add_token(type, value, line = @line) {|token| ... }
private
Yields a new token to the supplied block.
-
- (Object) advance(&block)
Advances through the input and generates the corresponding tokens.
-
- (Object) advance_column
private
-
- (Object) advance_line
private
-
- (Object) emit(type, start, stop, line = @line)
private
Emits a token of which the value is based on the supplied start/stop position.
-
- (Lexer) initialize(data, file = SourceLine::DEFAULT_FILE)
constructor
A new instance of Lexer.
-
- (Array) lex
Gathers all the tokens for the input and returns them as an Array.
-
- (Object) reset
Resets the internal state of the lexer.
-
- (String) slice_input(start, stop)
private
Returns the text between the specified start and stop position.
Constructor Details
- (Lexer) initialize(data, file = SourceLine::DEFAULT_FILE)
Returns a new instance of Lexer
202 203 204 205 206 207 |
# File 'lib/ll/lexer.rb', line 202 def initialize(data, file = SourceLine::DEFAULT_FILE) @data = data @file = file reset end |
Class Attribute Details
+ (Object) _ll_lexer_eof_trans (private)
Returns the value of attribute _ll_lexer_eof_trans
161 162 163 |
# File 'lib/ll/lexer.rb', line 161 def _ll_lexer_eof_trans @_ll_lexer_eof_trans end |
+ (Object) _ll_lexer_from_state_actions (private)
Returns the value of attribute _ll_lexer_from_state_actions
150 151 152 |
# File 'lib/ll/lexer.rb', line 150 def _ll_lexer_from_state_actions @_ll_lexer_from_state_actions end |
+ (Object) _ll_lexer_index_offsets (private)
Returns the value of attribute _ll_lexer_index_offsets
41 42 43 |
# File 'lib/ll/lexer.rb', line 41 def _ll_lexer_index_offsets @_ll_lexer_index_offsets end |
+ (Object) _ll_lexer_indicies (private)
Returns the value of attribute _ll_lexer_indicies
52 53 54 |
# File 'lib/ll/lexer.rb', line 52 def _ll_lexer_indicies @_ll_lexer_indicies end |
+ (Object) _ll_lexer_key_spans (private)
Returns the value of attribute _ll_lexer_key_spans
30 31 32 |
# File 'lib/ll/lexer.rb', line 30 def _ll_lexer_key_spans @_ll_lexer_key_spans end |
+ (Object) _ll_lexer_to_state_actions (private)
Returns the value of attribute _ll_lexer_to_state_actions
139 140 141 |
# File 'lib/ll/lexer.rb', line 139 def _ll_lexer_to_state_actions @_ll_lexer_to_state_actions end |
+ (Object) _ll_lexer_trans_actions (private)
Returns the value of attribute _ll_lexer_trans_actions
126 127 128 |
# File 'lib/ll/lexer.rb', line 126 def _ll_lexer_trans_actions @_ll_lexer_trans_actions end |
+ (Object) _ll_lexer_trans_keys (private)
Returns the value of attribute _ll_lexer_trans_keys
13 14 15 |
# File 'lib/ll/lexer.rb', line 13 def _ll_lexer_trans_keys @_ll_lexer_trans_keys end |
+ (Object) _ll_lexer_trans_targs (private)
Returns the value of attribute _ll_lexer_trans_targs
113 114 115 |
# File 'lib/ll/lexer.rb', line 113 def _ll_lexer_trans_targs @_ll_lexer_trans_targs end |
+ (Object) ll_lexer_en_main
Returns the value of attribute ll_lexer_en_main
189 190 191 |
# File 'lib/ll/lexer.rb', line 189 def ll_lexer_en_main @ll_lexer_en_main end |
+ (Object) ll_lexer_en_ruby_body
Returns the value of attribute ll_lexer_en_ruby_body
185 186 187 |
# File 'lib/ll/lexer.rb', line 185 def ll_lexer_en_ruby_body @ll_lexer_en_ruby_body end |
+ (Object) ll_lexer_error
Returns the value of attribute ll_lexer_error
180 181 182 |
# File 'lib/ll/lexer.rb', line 180 def ll_lexer_error @ll_lexer_error end |
+ (Object) ll_lexer_first_final
Returns the value of attribute ll_lexer_first_final
176 177 178 |
# File 'lib/ll/lexer.rb', line 176 def ll_lexer_first_final @ll_lexer_first_final end |
+ (Object) ll_lexer_start
Returns the value of attribute ll_lexer_start
172 173 174 |
# File 'lib/ll/lexer.rb', line 172 def ll_lexer_start @ll_lexer_start end |
Instance Method Details
- (Object) add_token(type, value, line = @line) {|token| ... } (private)
Yields a new token to the supplied block.
592 593 594 595 596 597 |
# File 'lib/ll/lexer.rb', line 592 def add_token(type, value, line = @line) source_line = SourceLine.new(@data, line, @column, @file) @column += value.length @block.call(Token.new(type, value, source_line)) end |
- (Object) advance(&block)
Advances through the input and generates the corresponding tokens. Each token is yielded to the supplied block.
240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 |
# File 'lib/ll/lexer.rb', line 240 def advance(&block) @block = block data = @data # saves ivar lookups while lexing. ts = nil te = nil cs = self.class.ll_lexer_start act = 0 eof = @data.bytesize p = 0 pe = eof mark = 0 brace_count = 0 start_line = 0 _ll_lexer_eof_trans = self.class.send(:_ll_lexer_eof_trans) _ll_lexer_from_state_actions = self.class.send(:_ll_lexer_from_state_actions) _ll_lexer_index_offsets = self.class.send(:_ll_lexer_index_offsets) _ll_lexer_indicies = self.class.send(:_ll_lexer_indicies) _ll_lexer_key_spans = self.class.send(:_ll_lexer_key_spans) _ll_lexer_to_state_actions = self.class.send(:_ll_lexer_to_state_actions) _ll_lexer_trans_actions = self.class.send(:_ll_lexer_trans_actions) _ll_lexer_trans_keys = self.class.send(:_ll_lexer_trans_keys) _ll_lexer_trans_targs = self.class.send(:_ll_lexer_trans_targs) # line 268 "lib/ll/lexer.rb" begin testEof = false _slen, _trans, _keys, _inds, _acts, _nacts = nil _goto_level = 0 _resume = 10 _eof_trans = 15 _again = 20 _test_eof = 30 _out = 40 while true if _goto_level <= 0 if p == pe _goto_level = _test_eof next end if cs == 0 _goto_level = _out next end end if _goto_level <= _resume case _ll_lexer_from_state_actions[cs] when 6 then # line 1 "NONE" begin ts = p end # line 296 "lib/ll/lexer.rb" end _keys = cs << 1 _inds = _ll_lexer_index_offsets[cs] _slen = _ll_lexer_key_spans[cs] _wide = ( (data.getbyte(p) || 0)) _trans = if ( _slen > 0 && _ll_lexer_trans_keys[_keys] <= _wide && _wide <= _ll_lexer_trans_keys[_keys + 1] ) then _ll_lexer_indicies[ _inds + _wide - _ll_lexer_trans_keys[_keys] ] else _ll_lexer_indicies[ _inds + _slen ] end end if _goto_level <= _eof_trans cs = _ll_lexer_trans_targs[_trans] if _ll_lexer_trans_actions[_trans] != 0 case _ll_lexer_trans_actions[_trans] when 25 then # line 188 "lib/ll/lexer.rl" begin te = p+1 begin brace_count += 1 end end when 26 then # line 190 "lib/ll/lexer.rl" begin te = p+1 begin if brace_count == 1 emit(:T_RUBY, mark, ts, start_line) mark = 0 brace_count = 0 start_line = 0 advance_column cs = 22; else brace_count -= 1 end end end when 23 then # line 206 "lib/ll/lexer.rl" begin te = p+1 end when 3 then # line 215 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_NAME, ts, te) end end when 4 then # line 216 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_TERMINALS, ts, te) end end when 2 then # line 217 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_INNER, ts, te) end end when 1 then # line 218 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_HEADER, ts, te) end end when 16 then # line 220 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_EQUALS, ts, te) end end when 14 then # line 221 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_COLON, ts, te) end end when 15 then # line 222 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_SEMICOLON, ts, te) end end when 20 then # line 223 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_PIPE, ts, te) end end when 13 then # line 225 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_PLUS, ts, te) end end when 12 then # line 226 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_STAR, ts, te) end end when 17 then # line 227 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_QUESTION, ts, te) end end when 10 then # line 228 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_LPAREN, ts, te) end end when 11 then # line 229 "lib/ll/lexer.rl" begin te = p+1 begin emit(:T_RPAREN, ts, te) end end when 19 then # line 231 "lib/ll/lexer.rl" begin te = p+1 begin mark = ts + 1 brace_count = 1 start_line = @line advance_column cs = 25; end end when 22 then # line 213 "lib/ll/lexer.rl" begin te = p p = p - 1; end when 21 then # line 1 "NONE" begin case act when 16 then begin begin p = ((te))-1; end emit(:T_EPSILON, ts, te) end when 23 then begin begin p = ((te))-1; end emit(:T_IDENT, ts, te) end end end when 24 then # line 148 "lib/ll/lexer.rl" begin advance_line @column = 1 end # line 186 "lib/ll/lexer.rl" begin te = p+1 end when 9 then # line 148 "lib/ll/lexer.rl" begin advance_line @column = 1 end # line 210 "lib/ll/lexer.rl" begin te = p+1 end when 8 then # line 154 "lib/ll/lexer.rl" begin advance_column end # line 211 "lib/ll/lexer.rl" begin te = p+1 end when 18 then # line 1 "NONE" begin te = p+1 end # line 224 "lib/ll/lexer.rl" begin act = 16; end when 7 then # line 1 "NONE" begin te = p+1 end # line 168 "lib/ll/lexer.rl" begin act = 23; end # line 507 "lib/ll/lexer.rb" end end end if _goto_level <= _again case _ll_lexer_to_state_actions[cs] when 5 then # line 1 "NONE" begin ts = nil; end # line 517 "lib/ll/lexer.rb" end if cs == 0 _goto_level = _out next end p += 1 if p != pe _goto_level = _resume next end end if _goto_level <= _test_eof if p == eof if _ll_lexer_eof_trans[cs] > 0 _trans = _ll_lexer_eof_trans[cs] - 1; _goto_level = _eof_trans next; end end end if _goto_level <= _out break end end end # line 81 "lib/ll/lexer.rl" # % fix highlight ensure reset end |
- (Object) advance_column (private)
603 604 605 |
# File 'lib/ll/lexer.rb', line 603 def advance_column @column += 1 end |
- (Object) advance_line (private)
599 600 601 |
# File 'lib/ll/lexer.rb', line 599 def advance_line @line += 1 end |
- (Object) emit(type, start, stop, line = @line) (private)
Emits a token of which the value is based on the supplied start/stop position.
566 567 568 569 570 |
# File 'lib/ll/lexer.rb', line 566 def emit(type, start, stop, line = @line) value = slice_input(start, stop) add_token(type, value, line) end |
- (Array) lex
Gathers all the tokens for the input and returns them as an Array.
215 216 217 218 219 220 221 222 223 |
# File 'lib/ll/lexer.rb', line 215 def lex tokens = [] advance do |token| tokens << token end return tokens end |
- (Object) reset
Resets the internal state of the lexer.
228 229 230 231 232 |
# File 'lib/ll/lexer.rb', line 228 def reset @block = nil @line = 1 @column = 1 end |
- (String) slice_input(start, stop) (private)
Returns the text between the specified start and stop position.
579 580 581 |
# File 'lib/ll/lexer.rb', line 579 def slice_input(start, stop) return @data.byteslice(start, stop - start) end |