PHPFixing
  • Privacy Policy
  • TOS
  • Ask Question
  • Contact Us
  • Home
  • PHP
  • Programming
  • SQL Injection
  • Web3.0

Thursday, September 15, 2022

[FIXED] How to speed up a repeated execution of exec()?

 September 15, 2022     exec, pypy, python     No comments   

Issue

I have the following code:

for k in pool:
    x = []
    y = []

    try:
        exec(pool[k])
    except Exception as e:
        ...
    do_something(x)
    do_something_else(y)

where pool[k] is python code that will eventually append items to x and y (that's why I am using exec instead of eval).

I have tried already to execute the same code with pypy but for this particular block I don't get much better, that line with exec is still my bottleneck.

That said, my question is: Is there a faster alternative to exec?

If not, do you have any workaround to get some speed up in such a case?

--UPDATE--

To clarify, pool contains around one million keys, to each key it is associated a script (around 50 line of code). The inputs for the scripts are defined before the for loop and the outputs generated by a script are stored in x and y. So, each script has a line in the code stating x.append(something) and y.append(something). The rest of the program will evaluate the results and score each script. Therefore, I need to loop over each script, execute it and process the results. The scripts are originally stored in different text files. pool is a dictionary obtained by parsing these files.

P.S. Using the pre-compiled version of the code:

for k in pool.keys():
    pool[k] = compile(pool[k], '<string>', 'exec')

I have got 5x speed increase, not much but it is something already. I am experimenting with other solutions...


Solution

If you really need to execute some code in such manner, use compile() to prepare it.

I.e. do not pass raw Python code into exec but compiled object. Use compile() on your codes before to make them Python byte compiled objects.

But still, it'll make more sense to write a function that will perform what you need on an input argument i.e. pool[k] and return results that are corresponding to x and y.

If you are getting your code out of a file, you have also IO slowdowns to cope with. So it would be nice to have these files already compiled to *.pyc.

You may think about using execfile() in Python2.

An idea for using functions in a pool:

template = """\
def newfunc ():
%s
    return result
"""

pool = [] # For iterating it will be faster if it is a list (just a bit)
# This compiles code as a real function and adds a pointer to a pool
def AddFunc (code):
    code = "\n".join(["    "+x for x in code.splitlines()])
    exec template % code
    pool.append(newfunc)

# Usage:
AddFunc("""\
a = 8.34**0.5
b = 8
c = 13
result = []
for x in range(10):
    result.append(math.sin(a*b+c)/math.pi+x)""")

for f in pool:
    x = f()


Answered By - Dalen
Answer Checked By - Senaida (PHPFixing Volunteer)
  • Share This:  
  •  Facebook
  •  Twitter
  •  Stumble
  •  Digg
Newer Post Older Post Home

0 Comments:

Post a Comment

Note: Only a member of this blog may post a comment.

Total Pageviews

Featured Post

Why Learn PHP Programming

Why Learn PHP Programming A widely-used open source scripting language PHP is one of the most popular programming languages in the world. It...

Subscribe To

Posts
Atom
Posts
Comments
Atom
Comments

Copyright © PHPFixing