Metaprogramming
Syntax#
- macro name(ex) … end
- quote … end
- :(…)
- $x
- Meta.quot(x)
- QuoteNode(x)
- esc(x)
Remarks#
Julia’s metaprogramming features are heavily inspired by those of Lisp-like languages, and will seem familiar to those with some Lisp background. Metaprogramming is very powerful. When used correctly, it can lead to more concise and readable code.
The quote ... end is quasiquote syntax. Instead of the expressions within being evaluated, they are simply parsed. The value of the quote ... end expression is the resulting Abstract Syntax Tree (AST).
The :(...) syntax is similar to the quote ... end syntax, but it is more lightweight. This syntax is more concise than quote ... end.
Inside a quasiquote, the $ operator is special and interpolates its argument into the AST. The argument is expected to be an expression which is spliced directly into the AST.
The Meta.quot(x) function quotes its argument. This is often useful in combination with using $ for interpolation, as it allows expressions and symbols to be spliced literally into the AST.
Reimplementing the @show macro
In Julia, the @show macro is often useful for debugging purposes. It displays both the expression to be evaluated and its result, finally returning the value of the result:
julia> @show 1 + 1
1 + 1 = 2
2It is straightforward to create our own version of @show:
julia> macro myshow(expression)
quote
value = $expression
println($(Meta.quot(expression)), " = ", value)
value
end
endTo use the new version, simply use the @myshow macro:
julia> x = @myshow 1 + 1
1 + 1 = 2
2
julia> x
2Until loop
We’re all used to the while syntax, that executes its body while the condition is evaluated to true. What if we want to implement an until loop, that executes a loop until the condition is evaluated to true?
In Julia, we can do this by creating a @until macro, that stops to execute its body when the condition is met:
macro until(condition, expression)
quote
while !($condition)
$expression
end
end |> esc
endHere we have used the function chaining syntax |>, which is equivalent to calling the esc function on the entire quote block. The esc function prevents macro hygiene from applying to the contents of the macro; without it, variables scoped in the macro will be renamed to prevent collisions with outside variables. See the Julia documentation on macro hygiene for more details.
You can use more than one expression in this loop, by simply putting everything inside a begin ... end block:
julia> i = 0;
julia> @until i == 10 begin
println(i)
i += 1
end
0
1
2
3
4
5
6
7
8
9
julia> i
10QuoteNode, Meta.quot, and Expr(:quote)
There are three ways to quote something using a Julia function:
julia> QuoteNode(:x)
:(:x)
julia> Meta.quot(:x)
:(:x)
julia> Expr(:quote, :x)
:(:x)What does “quoting” mean, and what is it good for? Quoting allows us to protect expressions from being interpreted as special forms by Julia. A common use case is when we generate expressions that should contain things that evaluate to symbols. (For example, this macro needs to return a expression that evaluates to a symbol.) It doesn’t work simply to return the symbol:
julia> macro mysym(); :x; end
@mysym (macro with 1 method)
julia> @mysym
ERROR: UndefVarError: x not defined
julia> macroexpand(:(@mysym))
:xWhat’s going on here? @mysym expands to :x, which as an expression becomes interpreted as the variable x. But nothing has been assigned to x yet, so we get an x not defined error.
To get around this, we must quote the result of our macro:
julia> macro mysym2(); Meta.quot(:x); end
@mysym2 (macro with 1 method)
julia> @mysym2
:x
julia> macroexpand(:(@mysym2))
:(:x)Here, we have used the Meta.quot function to turn our symbol into a quoted symbol, which is the result we want.
What is the difference between Meta.quot and QuoteNode, and which should I use? In almost all cases, the difference does not really matter. It is perhaps a little safer sometimes to use QuoteNode instead of Meta.quot. Exploring the difference is informative into how Julia expressions and macros work, however.
The difference between Meta.quot and QuoteNode, explained
Here’s a rule of thumb:
- If you need or want to support interpolation, use
Meta.quot; - If you can’t or don’t want to allow interpolation, use
QuoteNode.
In short, the difference is that Meta.quot allows interpolation within the quoted thing, while QuoteNode protects its argument from any interpolation. To understand interpolation, it is important to mention the $ expression. There is a kind of expression in Julia called a $ expression. These expressions allow for escaping. For instance, consider the following expression:
julia> ex = :( x = 1; :($x + $x) )
quote
x = 1
$(Expr(:quote, :($(Expr(:$, :x)) + $(Expr(:$, :x)))))
endWhen evaluated, this expression will evaluate 1 and assign it to x, then construct an expression of the form _ + _ where the _ will be replaced by the value of x. Thus, the result of this should be the expression 1 + 1 (which is not yet evaluated, and so distinct from the value 2). Indeed, this is the case:
julia> eval(ex)
:(1 + 1)Let’s say now that we’re writing a macro to build these kinds of expressions. Our macro will take an argument, which will replace the 1 in the ex above. This argument can be any expression, of course. Here is something that is not quite what we want:
julia> macro makeex(arg)
quote
:( x = $(esc($arg)); :($x + $x) )
end
end
@makeex (macro with 1 method)
julia> @makeex 1
quote
x = $(Expr(:escape, 1))
$(Expr(:quote, :($(Expr(:$, :x)) + $(Expr(:$, :x)))))
end
julia> @makeex 1 + 1
quote
x = $(Expr(:escape, 2))
$(Expr(:quote, :($(Expr(:$, :x)) + $(Expr(:$, :x)))))
endThe second case is incorrect, because we ought to keep 1 + 1 unevaluated. We fix that by quoting the argument with Meta.quot:
julia> macro makeex2(arg)
quote
:( x = $$(Meta.quot(arg)); :($x + $x) )
end
end
@makeex2 (macro with 1 method)
julia> @makeex2 1 + 1
quote
x = 1 + 1
$(Expr(:quote, :($(Expr(:$, :x)) + $(Expr(:$, :x)))))
endMacro hygiene does not apply to the contents of a quote, so escaping is not necessary in this case (and in fact not legal) in this case.
As mentioned earlier, Meta.quot allows interpolation. So let’s try that out:
julia> @makeex2 1 + $(sin(1))
quote
x = 1 + 0.8414709848078965
$(Expr(:quote, :($(Expr(:$, :x)) + $(Expr(:$, :x)))))
end
julia> let q = 0.5
@makeex2 1 + $q
end
quote
x = 1 + 0.5
$(Expr(:quote, :($(Expr(:$, :x)) + $(Expr(:$, :x)))))
endFrom the first example, we see that interpolation allows us to inline the sin(1), instead of having the expression be a literal sin(1). The second example shows that this interpolation is done in the macro invocation scope, not the macro’s own scope. That’s because our macro hasn’t actually evaluated any code; all it’s doing is generating code. The evaluation of the code (which makes its way into the expression) is done when the expression the macro generates is actually run.
What if we had used QuoteNode instead? As you may guess, since QuoteNode prevents interpolation from happening at all, this means it won’t work.
julia> macro makeex3(arg)
quote
:( x = $$(QuoteNode(arg)); :($x + $x) )
end
end
@makeex3 (macro with 1 method)
julia> @makeex3 1 + $(sin(1))
quote
x = 1 + $(Expr(:$, :(sin(1))))
$(Expr(:quote, :($(Expr(:$, :x)) + $(Expr(:$, :x)))))
end
julia> let q = 0.5
@makeex3 1 + $q
end
quote
x = 1 + $(Expr(:$, :q))
$(Expr(:quote, :($(Expr(:$, :x)) + $(Expr(:$, :x)))))
end
julia> eval(@makeex3 $(sin(1)))
ERROR: unsupported or misplaced expression $
in eval(::Module, ::Any) at ./boot.jl:234
in eval(::Any) at ./boot.jl:233In this example, we might agree that Meta.quot gives greater flexibility, as it allows interpolation. So why might we ever consider using QuoteNode? In some cases, we may not actually desire interpolation, and actually want the literal $ expression. When would that be desirable? Let’s consider a generalization of @makeex where we can pass additional arguments determining what comes to the left and right of the + sign:
julia> macro makeex4(expr, left, right)
quote
quote
$$(Meta.quot(expr))
:($$$(Meta.quot(left)) + $$$(Meta.quot(right)))
end
end
end
@makeex4 (macro with 1 method)
julia> @makeex4 x=1 x x
quote # REPL[110], line 4:
x = 1 # REPL[110], line 5:
$(Expr(:quote, :($(Expr(:$, :x)) + $(Expr(:$, :x)))))
end
julia> eval(ans)
:(1 + 1)A limitation of our implementation of @makeex4 is that we can’t use expressions as either the left and right sides of the expression directly, because they get interpolated. In other words, the expressions may get evaluated for interpolation, but we might want them preserved. (Since there are many levels of quoting and evaluation here, let us clarify: our macro generates code that constructs an expression that when evaluated produces another expression. Phew!)
julia> @makeex4 x=1 x/2 x
quote # REPL[110], line 4:
x = 1 # REPL[110], line 5:
$(Expr(:quote, :($(Expr(:$, :(x / 2))) + $(Expr(:$, :x)))))
end
julia> eval(ans)
:(0.5 + 1)We ought to allow the user to specify when interpolation is to happen, and when it shouldn’t. Theoretically, that’s an easy fix: we can just remove one of the $ signs in our application, and let the user contribute their own. What this means is that we interpolate a quoted version of the expression entered by the user (which we’ve already quoted and interpolated once). This leads to the following code, which can be a little confusing at first, due to the multiple nested levels of quoting and unquoting. Try to read and understand what each escape is for.
julia> macro makeex5(expr, left, right)
quote
quote
$$(Meta.quot(expr))
:($$(Meta.quot($(Meta.quot(left)))) + $$(Meta.quot($(Meta.quot(right)))))
end
end
end
@makeex5 (macro with 1 method)
julia> @makeex5 x=1 1/2 1/4
quote # REPL[121], line 4:
x = 1 # REPL[121], line 5:
$(Expr(:quote, :($(Expr(:$, :($(Expr(:quote, :(1 / 2)))))) + $(Expr(:$, :($(Expr(:quote, :(1 / 4)))))))))
end
julia> eval(ans)
:(1 / 2 + 1 / 4)
julia> @makeex5 y=1 $y $y
ERROR: UndefVarError: y not definedThings started well, but something has gone wrong. The macro’s generated code is trying to interpolate the copy of y in the macro invocation scope; but there is no copy of y in the macro invocation scope. Our error is allowing interpolation with the second and third arguments in the macro. To fix this error, we must use QuoteNode.
julia> macro makeex6(expr, left, right)
quote
quote
$$(Meta.quot(expr))
:($$(Meta.quot($(QuoteNode(left)))) + $$(Meta.quot($(QuoteNode(right)))))
end
end
end
@makeex6 (macro with 1 method)
julia> @makeex6 y=1 1/2 1/4
quote # REPL[129], line 4:
y = 1 # REPL[129], line 5:
$(Expr(:quote, :($(Expr(:$, :($(Expr(:quote, :(1 / 2)))))) + $(Expr(:$, :($(Expr(:quote, :(1 / 4)))))))))
end
julia> eval(ans)
:(1 / 2 + 1 / 4)
julia> @makeex6 y=1 $y $y
quote # REPL[129], line 4:
y = 1 # REPL[129], line 5:
$(Expr(:quote, :($(Expr(:$, :($(Expr(:quote, :($(Expr(:$, :y)))))))) + $(Expr(:$, :($(Expr(:quote, :($(Expr(:$, :y)))))))))))
end
julia> eval(ans)
:(1 + 1)
julia> @makeex6 y=1 1+$y $y
quote # REPL[129], line 4:
y = 1 # REPL[129], line 5:
$(Expr(:quote, :($(Expr(:$, :($(Expr(:quote, :(1 + $(Expr(:$, :y)))))))) + $(Expr(:$, :($(Expr(:quote, :($(Expr(:$, :y)))))))))))
end
julia> @makeex6 y=1 $y/2 $y
quote # REPL[129], line 4:
y = 1 # REPL[129], line 5:
$(Expr(:quote, :($(Expr(:$, :($(Expr(:quote, :($(Expr(:$, :y)) / 2)))))) + $(Expr(:$, :($(Expr(:quote, :($(Expr(:$, :y)))))))))))
end
julia> eval(ans)
:(1 / 2 + 1)By using QuoteNode, we have protected our arguments from interpolation. Since QuoteNode only has the effect of additional protections, it is never harmful to use QuoteNode, unless you desire interpolation. However, understanding the difference makes it possible to understand where and why Meta.quot could be a better choice.
This long exercise is with an example that is plainly too complex to show up in any reasonable application. Therefore, we have justified the following rule of thumb, mentioned earlier:
- If you need or want to support interpolation, use
Meta.quot; - If you can’t or don’t want to allow interpolation, use
QuoteNode.
What about Expr(:quote)?
Expr(:quote, x) is equivalent to Meta.quot(x). However, the latter is more idiomatic and is preferred. For code that heavily uses metaprogramming, a using Base.Meta line is often used, which allows Meta.quot to be referred to as simply quot.
Guide
π’s Metaprogramming bits & bobs
Goals:
-
Teach through minimal targeted functional/useful/non-abstract examples (e.g.
@swapor@assert) that introduce concepts in suitable contexts -
Prefer to let the code illustrate/demonstrate the concepts rather than paragraphs of explanation
-
Avoid linking ‘required reading’ to other pages — it interrupts the narrative
-
Present things in a sensible order that will making learning easiest
Resources:
julialang.org
wikibook (@Cormullion)
5 layers (Leah Hanson)
SO-Doc Quoting (@TotalVerb)
SO-Doc — Symbols that are not legal identifiers (@TotalVerb)
SO: What is a Symbol in Julia (@StefanKarpinski)
Discourse thread (@p-i-)
https://stackoverflow.com/documentation/julia-lang/1945/metaprogramming#t=201612021739596755987
Most of the material has come from the discourse channel, most of that has come from fcard… please prod me if I had forgotten attributions.
Symbol
julia> mySymbol = Symbol("myName") # or 'identifier'
:myName
julia> myName = 42
42
julia> mySymbol |> eval # 'foo |> bar' puts output of 'foo' into 'bar', so 'bar(foo)'
42
julia> :( $mySymbol = 1 ) |> eval
1
julia> myName
1Passing flags into functions:
function dothing(flag)
if flag == :thing_one
println("did thing one")
elseif flag == :thing_two
println("did thing two")
end
end
julia> dothing(:thing_one)
did thing one
julia> dothing(:thing_two)
did thing twoA hashkey example:
number_names = Dict{Symbol, Int}()
number_names[:one] = 1
number_names[:two] = 2
number_names[:six] = 6(Advanced)
(@fcard) :foo a.k.a. :(foo) yields a symbol if foo is a valid identifier, otherwise an expression.
# NOTE: Different use of ':' is:
julia> :mySymbol = Symbol('hello world')
#(You can create a symbol with any name with Symbol("<name>"),
# which lets us create such gems as:
julia> one_plus_one = Symbol("1 + 1")
Symbol("1 + 1")
julia> eval(one_plus_one)
ERROR: UndefVarError: 1 + 1 not defined
...
julia> valid_math = :($one_plus_one = 3)
:(1 + 1 = 3)
julia> one_plus_one_plus_two = :($one_plus_one + 2)
:(1 + 1 + 2)
julia> eval(quote
$valid_math
@show($one_plus_one_plus_two)
end)
1 + 1 + 2 = 5
...Basically you can treat Symbols as lightweight strings. That’s not what they’re for, but you can do it, so why not. Julia’s Base itself does it, print_with_color(:red, "abc") prints a red-colored abc .
Expr (AST)
(Almost) everything in Julia is an expression, i.e. an instance of Expr, which will hold an AST.
# when you type ...
julia> 1+1
2
# Julia is doing: eval(parse("1+1"))
# i.e. First it parses the string "1+1" into an `Expr` object ...
julia> ast = parse("1+1")
:(1 + 1)
# ... which it then evaluates:
julia> eval(ast)
2
# An Expr instance holds an AST (Abstract Syntax Tree). Let's look at it:
julia> dump(ast)
Expr
head: Symbol call
args: Array{Any}((3,))
1: Symbol +
2: Int64 1
3: Int64 1
typ: Any
# TRY: fieldnames(typeof(ast))
julia> :(a + b*c + 1) ==
parse("a + b*c + 1") ==
Expr(:call, :+, :a, Expr(:call, :*, :b, :c), 1)
trueNesting Exprs:
julia> dump( :(1+2/3) )
Expr
head: Symbol call
args: Array{Any}((3,))
1: Symbol +
2: Int64 1
3: Expr
head: Symbol call
args: Array{Any}((3,))
1: Symbol /
2: Int64 2
3: Int64 3
typ: Any
typ: Any
# Tidier rep'n using s-expr
julia> Meta.show_sexpr( :(1+2/3) )
(:call, :+, 1, (:call, :/, 2, 3))multiline Exprs using quote
julia> blk = quote
x=10
x+1
end
quote # REPL[121], line 2:
x = 10 # REPL[121], line 3:
x + 1
end
julia> blk == :( begin x=10; x+1 end )
true
# Note: contains debug info:
julia> Meta.show_sexpr(blk)
(:block,
(:line, 2, Symbol("REPL[121]")),
(:(=), :x, 10),
(:line, 3, Symbol("REPL[121]")),
(:call, :+, :x, 1)
)
# ... unlike:
julia> noDbg = :( x=10; x+1 )
quote
x = 10
x + 1
end… so quote is functionally the same but provides extra debug info.
(*) TIP: Use let to keep x within the block
quote -ing a quote
Expr(:quote, x) is used to represent quotes within quotes.
Expr(:quote, :(x + y)) == :(:(x + y))
Expr(:quote, Expr(:$, :x)) == :(:($x))QuoteNode(x) is similar to Expr(:quote, x) but it prevents interpolation.
eval(Expr(:quote, Expr(:$, 1))) == 1
eval(QuoteNode(Expr(:$, 1))) == Expr(:$, 1)Are $ and :(…) somehow inverses of one another?
:(foo) means “don’t look at the value, look at the expression”
$foo means “change the expression to its value”
:($(foo)) == foo.
$(:(foo)) is an error.
$(...) isn’t an operation and doesn’t do anything by itself, it’s an “interpolate this!” sign that the quoting syntax uses. i.e. It only exists within a quote.
Is $foo the same as eval(foo) ?
No!
$foo is exchanged for the compile-time value
eval(foo) means to do that at runtime
eval will occur in the global scope
interpolation is local
eval(:<expr>) should return the same as just <expr> (assuming <expr> is a valid expression in the current global space)
eval(:(1 + 2)) == 1 + 2
eval(:(let x=1; x + 1 end)) == let x=1; x + 1 endmacro s
Ready? :)
# let's try to make this!
julia> x = 5; @show x;
x = 5Let’s make our own @show macro:
macro log(x)
:(
println( "Expression: ", $(string(x)), " has value: ", $x )
)
end
u = 42
f = x -> x^2
@log(u) # Expression: u has value: 42
@log(42) # Expression: 42 has value: 42
@log(f(42)) # Expression: f(42) has value: 1764
@log(:u) # Expression: :u has value: uexpand to lower an Expr
5 layers (Leah Hanson) <— explains how Julia takes source code as a string, tokenizes it into an Expr-tree (AST), expands out all the macros (still AST), lowers (lowered AST), then converts into LLVM (and beyond — at the moment we don’t need to worry what lies beyond!)
Q: code_lowered acts on functions. Is it possible to lower an Expr?
A: yup!
# function -> lowered-AST
julia> code_lowered(*,(String,String))
1-element Array{LambdaInfo,1}:
LambdaInfo template for *(s1::AbstractString, ss::AbstractString...) at strings/basic.jl:84
# Expr(i.e. AST) -> lowered-AST
julia> expand(:(x ? y : z))
:(begin
unless x goto 3
return y
3:
return z
end)
julia> expand(:(y .= x.(i)))
:((Base.broadcast!)(x,y,i))
# 'Execute' AST or lowered-AST
julia> eval(ast)If you want to only expand macros you can use macroexpand:
# AST -> (still nonlowered-)AST but with macros expanded:
julia> macroexpand(:(@show x))
quote
(Base.println)("x = ",(Base.repr)(begin # show.jl, line 229:
#28#value = x
end))
#28#value
end…which returns a non-lowered AST but with all macros expanded.
esc()
esc(x) returns an Expr that says “don’t apply hygiene to this”, it’s the same as Expr(:escape, x). Hygiene is what keeps a macro self-contained, and you esc things if you want them to “leak”. e.g.
Example: swap macro to illustrate esc()
macro swap(p, q)
quote
tmp = $(esc(p))
$(esc(p)) = $(esc(q))
$(esc(q)) = tmp
end
end
x,y = 1,2
@swap(x,y)
println(x,y) # 2 1$ allows us to ‘escape out of’ the quote.
So why not simply $p and $q? i.e.
# FAIL!
tmp = $p
$p = $q
$q = tmpBecause that would look first to the macro scope for p, and it would find a local p i.e. the parameter p (yes, if you subsequently access p without esc-ing, the macro considers the p parameter as a local variable).
So $p = ... is just a assigning to the local p. it’s not affecting whatever variable was passed-in in the calling context.
Ok so how about:
# Almost!
tmp = $p # <-- you might think we don't
$(esc(p)) = $q # need to esc() the RHS
$(esc(q)) = tmpSo esc(p) is ‘leaking’ p into the calling context. “The thing that was passed into the macro that we receive as p”
julia> macro swap(p, q)
quote
tmp = $p
$(esc(p)) = $q
$(esc(q)) = tmp
end
end
@swap (macro with 1 method)
julia> x, y = 1, 2
(1,2)
julia> @swap(x, y);
julia> @show(x, y);
x = 2
y = 1
julia> macroexpand(:(@swap(x, y)))
quote # REPL[34], line 3:
#10#tmp = x # REPL[34], line 4:
x = y # REPL[34], line 5:
y = #10#tmp
end As you can see tmp gets the hygiene treatment #10#tmp, whereas x and y don’t.
Julia is making a unique identifier for tmp, something you can manually do with gensym, ie:
julia> gensym(:tmp)
Symbol("##tmp#270")But: There is a gotcha:
julia> module Swap
export @swap
macro swap(p, q)
quote
tmp = $p
$(esc(p)) = $q
$(esc(q)) = tmp
end
end
end
Swap
julia> using Swap
julia> x,y = 1,2
(1,2)
julia> @swap(x,y)
ERROR: UndefVarError: x not definedAnother thing julia’s macro hygiene does is, if the macro is from another module, it makes any variables (that were not assigned inside the macro’s returning expression, like tmp in this case) globals of the current module, so $p becomes Swap.$p, likewise $q -> Swap.$q.
In general, if you need a variable that is outside the macro’s scope you should esc it, so you should esc(p) and esc(q) regardless if they are on the LHS or RHS of a expression, or even by themselves.
people have already mentioned gensyms a few times and soon you will be seduced by the dark side of defaulting to escaping the whole expression with a few gensyms peppered here and there, but… Make sure to understand how hygiene works before trying to be smarter than it! It’s not a particularly complex algorithm so it shouldn’t take too long, but don’t rush it! Don’t use that power until you understand all the ramifications of it… (@fcard)
Example: until macro
(@Ismael-VC)
"until loop"
macro until(condition, block)
quote
while ! $condition
$block
end
end |> esc
end
julia> i=1; @until( i==5, begin; print(i); i+=1; end )
1234
(@fcard) |> is controversial, however. I am surprised a mob hasn’t come to argue yet. (maybe everyone is just tired of it). There is a recommendation of having most if not all of the macro just be a call to a function, so:
macro until(condition, block)
esc(until(condition, block))
end
function until(condition, block)
quote
while !$condition
$block
end
end
end…is a safer alternative.
##@fcard’s simple macro challenge
Task: Swap the operands, so swaps(1/2) gives 2.00 i.e. 2/1
macro swaps(e)
e.args[2:3] = e.args[3:-1:2]
e
end
@swaps(1/2)
2.00More macro challenges from @fcard here
Interpolation and assert macro
https://docs.julialang.org/en/release-0.5/manual/metaprogramming/#building-an-advanced-macro
macro assert(ex)
return :( $ex ? nothing : throw(AssertionError($(string(ex)))) )
endQ: Why the last $?
A: It interpolates, i.e. forces Julia to eval that string(ex) as execution passes through the invocation of this macro.
i.e. If you just run that code it won’t force any evaluation. But the moment you do assert(foo) Julia will invoke this macro replacing its ‘AST token/Expr’ with whatever it returns, and the $ will kick into action.
A fun hack for using { } for blocks
(@fcard) I don’t think there is anything technical keeping {} from being used as blocks, in fact one can even pun on the residual {} syntax to make it work:
julia> macro c(block)
@assert block.head == :cell1d
esc(quote
$(block.args...)
end)
end
@c (macro with 1 method)
julia> @c {
print(1)
print(2)
1+2
}
123*(unlikely to still work if/when the {} syntax is repurposed)
So first Julia sees the macro token, so it will read/parse tokens until the matching end, and create what? An Expr with .head=:macro or something? Does it store "a+1" as a string or does it break it apart into :+(:a, 1)? How to view?
?
(@fcard) In this case because of lexical scope, a is undefined in @Ms scope so it uses the global variable…
I actually forgot to escape the flipplin’ expression in my dumb example, but the “only works within the same module” part of it still applies.
julia> module M
macro m()
:(a+1)
end
end
M
julia> a = 1
1
julia> M.@m
ERROR: UndefVarError: a not definedThe reason being that, if the macro is used in any module other than the one it was defined in, any variables not defined within the code-to-be-expanded are treated as globals of the macro’s module.
julia> macroexpand(:(M.@m))
:(M.a + 1)ADVANCED
###@Ismael-VC
@eval begin
"do-until loop"
macro $(:do)(block, until::Symbol, condition)
until ≠ :until &&
error("@do expected `until` got `$until`")
quote
let
$block
@until $condition begin
$block
end
end
end |> esc
end
end
julia> i = 0
0
julia> @do begin
@show i
i += 1
end until i == 5
i = 0
i = 1
i = 2
i = 3
i = 4Scott’s macro:
"""
Internal function to return captured line number information from AST
##Parameters
- a: Expression in the julia type Expr
##Return
- Line number in the file where the calling macro was invoked
"""
_lin(a::Expr) = a.args[2].args[1].args[1]
"""
Internal function to return captured file name information from AST
##Parameters
- a: Expression in the julia type Expr
##Return
- The name of the file where the macro was invoked
"""
_fil(a::Expr) = string(a.args[2].args[1].args[2])
"""
Internal function to determine if a symbol is a status code or variable
"""
function _is_status(sym::Symbol)
sym in (:OK, :WARNING, :ERROR) && return true
str = string(sym)
length(str) > 4 && (str[1:4] == "ERR_" || str[1:5] == "WARN_" || str[1:5] == "INFO_")
end
"""
Internal function to return captured error code from AST
##Parameters
- a: Expression in the julia type Expr
##Return
- Error code from the captured info in the AST from the calling macro
"""
_err(a::Expr) =
(sym = a.args[2].args[2] ; _is_status(sym) ? Expr(:., :Status, QuoteNode(sym)) : sym)
"""
Internal function to produce a call to the log function based on the macro arguments and the AST from the ()->ERRCODE anonymous function definition used to capture error code, file name and line number where the macro is used
##Parameters
- level: Loglevel which has to be logged with macro
- a: Expression in the julia type Expr
- msgs: Optional message
##Return
- Statuscode
"""
function _log(level, a, msgs)
if isempty(msgs)
:( log($level, $(esc(:Symbol))($(_fil(a))), $(_lin(a)), $(_err(a)) )
else
:( log($level, $(esc(:Symbol))($(_fil(a))), $(_lin(a)), $(_err(a)), message=$(esc(msgs[1]))) )
end
end
macro warn(a, msgs...) ; _log(Warning, a, msgs) ; endjunk / unprocessed …
view/dump a macro
(@p-i-) Suppose I just do macro m(); a+1; end in a fresh REPL. With no a defined. How can I ‘view’ it?
like, is there some way to ‘dump’ a macro?
Without actually executing it
(@fcard) All the code in macros are actually put into functions, so you can only view their lowered or type-inferred code.
julia> macro m() a+1 end
@m (macro with 1 method)
julia> @code_typed @m
LambdaInfo for @m()
:(begin
return Main.a + 1
end)
julia> @code_lowered @m
CodeInfo(:(begin
nothing
return Main.a + 1
end))
# ^ or: code_lowered(eval(Symbol("@m")))[1] # ouf!Other ways to get a macro’s function:
julia> macro getmacro(call) call.args[1] end
@getmacro (macro with 1 method)
julia> getmacro(name) = getfield(current_module(), name.args[1])
getmacro (generic function with 1 method)
julia> @getmacro @m
@m (macro with 1 method)
julia> getmacro(:@m)
@m (macro with 1 method)julia> eval(Symbol("@M"))
@M (macro with 1 method)
julia> dump( eval(Symbol("@M")) )
@M (function of type #@M)
julia> code_typed( eval(Symbol("@M")) )
1-element Array{Any,1}:
LambdaInfo for @M()
julia> code_typed( eval(Symbol("@M")) )[1]
LambdaInfo for @M()
:(begin
return $(Expr(:copyast, :($(QuoteNode(:(a + 1))))))
end::Expr)
julia> @code_typed @M
LambdaInfo for @M()
:(begin
return $(Expr(:copyast, :($(QuoteNode(:(a + 1))))))
end::Expr)^ looks like I can use code_typed instead
How to understand eval(Symbol("@M"))?
(@fcard) Currently, every macro has a function associated with it. If you have a macro called M, then the macro’s function is called @M. Generally you can get a function’s value with e.g. eval(:print) but with a macro’s function you need to do Symbol("@M"), since just :@M becomes an Expr(:macrocall, Symbol("@M")) and evaluating that causes a macro-expansion.
Why doesn’t code_typed display params?
(@p-i-)
julia> code_typed( x -> x^2 )[1]
LambdaInfo for (::##5#6)(::Any)
:(begin
return x ^ 2
end)^ here I see one ::Any param, but it doesn’t seem to be connected with the token x.
julia> code_typed( print )[1]
LambdaInfo for print(::IO, ::Char)
:(begin
(Base.write)(io,c)
return Base.nothing
end::Void)^ similarly here; there is nothing to connect io with the ::IO
So surely this can’t be a complete dump of the AST representation of that particular print method…?
(@fcard) print(::IO, ::Char) only tells you what method it is, it’s not part of the AST.
It isn’t even present in master anymore:
julia> code_typed(print)[1]
CodeInfo(:(begin
(Base.write)(io,c)
return Base.nothing
end))=>Void(@p-i-) I don’t understand what you mean by that. It seems to be dumping the AST for the body of that method, no?
I thought code_typed gives the AST for a function. But it seems to be missing the first step, i.e. setting up tokens for params.
(@fcard) code_typed is meant to only show the body’s AST, but for now it does give the complete AST of the method, in the form of a LambdaInfo (0.5) or CodeInfo (0.6), but a lot of the information is omitted when printed to the repl. You will need to inspect the LambdaInfo field by field in order to get all the details. dump is going to flood your repl, so you could try:
macro method_info(call)
quote
method = @code_typed $(esc(call))
print_info_fields(method)
end
end
function print_info_fields(method)
for field in fieldnames(typeof(method))
if isdefined(method, field) && !(field in [Symbol(""), :code])
println(" $field = ", getfield(method, field))
end
end
display(method)
end
print_info_fields(x::Pair) = print_info_fields(x[1])Which gives all the values of the named fields of a method’s AST:
julia> @method_info print(STDOUT, 'a')
rettype = Void
sparam_syms = svec()
sparam_vals = svec()
specTypes = Tuple{Base.#print,Base.TTY,Char}
slottypes = Any[Base.#print,Base.TTY,Char]
ssavaluetypes = Any[]
slotnames = Any[Symbol("#self#"),:io,:c]
slotflags = UInt8[0x00,0x00,0x00]
def = print(io::IO, c::Char) at char.jl:45
nargs = 3
isva = false
inferred = true
pure = false
inlineable = true
inInference = false
inCompile = false
jlcall_api = 0
fptr = Ptr{Void} @0x00007f7a7e96ce10
LambdaInfo for print(::Base.TTY, ::Char)
:(begin
$(Expr(:invoke, LambdaInfo for write(::Base.TTY, ::Char), :(Base.write), :(io), :(c)))
return Base.nothing
end::Void)See the lil’ def = print(io::IO, c::Char)? There you go! (also the slotnames = [..., :io, :c] part)
Also yes, the difference in output is because I was showing the results on master.
???
(@Ismael-VC) you mean like this? https://stackoverflow.com/questions/39314925/generic-dispatch-with-symbols/39315024#39315024
You can do it this way:
julia> function dispatchtest{alg}(::Type{Val{alg}})
println("This is the generic dispatch. The algorithm is $alg")
end
dispatchtest (generic function with 1 method)
julia> dispatchtest(alg::Symbol) = dispatchtest(Val{alg})
dispatchtest (generic function with 2 methods)
julia> function dispatchtest(::Type{Val{:Euler}})
println("This is for the Euler algorithm!")
end
dispatchtest (generic function with 3 methods)
julia> dispatchtest(:Foo)
This is the generic dispatch. The algorithm is Foo
julia> dispatchtest(:Euler)This is for the Euler algorithm! I wonder what does @fcard thinks about generic symbol dispatch! ---^ :angel:
Module Gotcha
@def m begin
a+2
end
@m # replaces the macro at compile-time with the expression a+2More accurately, only works within the toplevel of the module the macro was defined in.
julia> module M
macro m1()
a+1
end
end
M
julia> macro m2()
a+1
end
@m2 (macro with 1 method)
julia> a = 1
1
julia> M.@m1
ERROR: UndefVarError: a not defined
julia> @m2
2
julia> let a = 20
@m2
end
2esc keeps this from happening, but defaulting to always using it goes against the language design.
A good defense for this is to keep one from using and introducing names within macros, which makes them hard to track to a human reader.
Python dict/JSON like syntax for Dict literals.
Introduction
Julia uses the following syntax for dictionaries:
Dict({k₁ => v₁, k₂ => v₂, …, kₙ₋₁ => vₙ₋₁, kₙ => vₙ)While Python and JSON looks like this:
{k₁: v₁, k₂: v₂, …, kₙ₋₁: vₙ₋₁, kₙ: vₙ}For illustrative purposes we could also use this syntax in Julia and add new semantics to it (Dict syntax is the idiomatic way in Julia, which is recommended).
First let’s see what kind of expression it is:
julia> parse("{1:2 , 3: 4}") |> Meta.show_sexpr
(:cell1d, (:(:), 1, 2), (:(:), 3, 4))This means we need to take this :cell1d expression and either transform it or return a new expression that should look like this:
julia> parse("Dict(1 => 2 , 3 => 4)") |> Meta.show_sexpr
(:call, :Dict, (:(=>), 1, 2), (:(=>), 3, 4))Macro definition
The following macro, while simple, allows to demonstrate such code generation and transformation:
macro dict(expr)
# Check the expression has the correct form:
if expr.head ≠ :cell1d || any(sub_expr.head ≠ :(:) for sub_expr ∈ expr.args)
error("syntax: expected `{k₁: v₁, k₂: v₂, …, kₙ₋₁: vₙ₋₁, kₙ: vₙ}`")
end
# Create empty `:Dict` expression which will be returned:
block = Expr(:call, :Dict) # :(Dict())
# Append `(key => value)` pairs to the block:
for pair in expr.args
k, v = pair.args
push!(block.args, :($k => $v))
end # :(Dict(k₁ => v₁, k₂ => v₂, …, kₙ₋₁ => vₙ₋₁, kₙ => vₙ))
# Block is escaped so it can reach variables from it's calling scope:
return esc(block)
endLet’s check out the resulting macro expansion:
julia> :(@dict {"a": :b, 'c': 1, :d: 2.0}) |> macroexpand
:(Dict("a" => :b,'c' => 1,:d => 2.0))Usage
julia> @dict {"a": :b, 'c': 1, :d: 2.0}
Dict{Any,Any} with 3 entries:
"a" => :b
:d => 2.0
'c' => 1
julia> @dict {
"string": :b,
'c' : 1,
:symbol : π,
Function: print,
(1:10) : range(1, 10)
}
Dict{Any,Any} with 5 entries:
1:10 => 1:10
Function => print
"string" => :b
:symbol => π = 3.1415926535897...
'c' => 1 The last example is exactly equivalent to:
Dict(
"string" => :b,
'c' => 1,
:symbol => π,
Function => print,
(1:10) => range(1, 10)
)Misusage
julia> @dict {"one": 1, "two": 2, "three": 3, "four": 4, "five" => 5}
syntax: expected `{k₁: v₁, k₂: v₂, …, kₙ₋₁: vₙ₋₁, kₙ: vₙ}`
julia> @dict ["one": 1, "two": 2, "three": 3, "four": 4, "five" => 5]
syntax: expected `{k₁: v₁, k₂: v₂, …, kₙ₋₁: vₙ₋₁, kₙ: vₙ}`Notice that Julia has other uses for colon : as such you will need to wrap range literal expressions with parenthesis or use the range function, for example.