• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

Python conf.combine_lists函数代码示例

原作者: [db:作者] 来自: [db:来源] 收藏 邀请

本文整理汇总了Python中mrjob.conf.combine_lists函数的典型用法代码示例。如果您正苦于以下问题:Python combine_lists函数的具体用法?Python combine_lists怎么用?Python combine_lists使用的例子?那么恭喜您, 这里精选的函数代码示例或许可以为您提供帮助。



在下文中一共展示了combine_lists函数的11个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Python代码示例。

示例1: libjars

    def libjars(self):
        """Optional list of paths of jar files to run our job with using
        Hadoop's ``-libjar`` option. Normally setting :py:attr:`LIBJARS`
        is sufficient.

        By default, this combines :option:`libjars` options from the command
        lines with :py:attr:`LIBJARS`, with command line arguments taking
        precedence. Paths from :py:attr:`LIBJARS` are interpreted as relative
        to the the directory containing the script (paths from the
        command-line are relative to the current working directory).

        Note that ``~`` and environment variables in paths will always be
        expanded by the job runner (see :mrjob-opt:`libjars`).

        .. versionadded:: 0.5.3
        """
        script_dir = os.path.dirname(self.mr_job_script())

        paths_from_libjars = []

        # libjar paths will eventually be combined with combine_path_lists,
        # which will expand environment variables. We don't want to assume
        # a path like $MY_DIR/some.jar is always relative ($MY_DIR could start
        # with /), but we also don't want to expand environment variables
        # prematurely.
        for path in self.LIBJARS or []:
            if os.path.isabs(expand_path(path)):
                paths_from_libjars.append(path)
            else:
                paths_from_libjars.append(os.path.join(script_dir, path))

        return combine_lists(paths_from_libjars, self.options.libjars)
开发者ID:etiennebatise,项目名称:mrjob,代码行数:32,代码来源:job.py


示例2: _job_kwargs

 def _job_kwargs(self):
     """Keyword arguments to the runner class that can be specified
     by the job/launcher itself."""
     # use the most basic combiners; leave magic like resolving paths
     # and blanking out jobconf values to the runner
     return dict(
         # command-line has the final say on jobconf and libjars
         jobconf=combine_dicts(
             self.jobconf(), self.options.jobconf),
         libjars=combine_lists(
             self.libjars(), self.options.libjars),
         partitioner=self.partitioner(),
         sort_values=self.sort_values(),
         # TODO: should probably put self.options last below for consistency
         upload_archives=combine_lists(
             self.options.upload_archives, self.archives()),
         upload_dirs=combine_lists(
             self.options.upload_dirs, self.dirs()),
         upload_files=combine_lists(
             self.options.upload_files, self.files()),
     )
开发者ID:Yelp,项目名称:mrjob,代码行数:21,代码来源:launch.py


示例3: emr_job_runner_kwargs

 def emr_job_runner_kwargs(self):
     args = super(DownloadToS3, self).emr_job_runner_kwargs()
     
     # set up AWS credentials on EMR instances
     access_key = os.environ['AWS_ACCESS_KEY_ID']
     secret = os.environ['AWS_SECRET_ACCESS_KEY']
     args['cmdenv'] = combine_dicts(args['cmdenv'], {'AWS_ACCESS_KEY_ID': access_key, 'AWS_SECRET_ACCESS_KEY': secret})
     
     # install pip, aws-cli, and boto
     args['bootstrap_cmds'] = combine_lists(args['bootstrap_cmds'],
                                            ['sysctl -w "net.ipv4.tcp_window_scaling=0"',
                                             'sudo apt-get install python-pip',
                                             'sudo pip install awscli',
                                             'sudo pip install boto'])
     return args
开发者ID:laserson,项目名称:hadoop-genomics-demo,代码行数:15,代码来源:download_to_s3.py


示例4: test_concatenation

 def test_concatenation(self):
     assert_equal(combine_lists([1, 2], None, (3, 4)), [1, 2, 3, 4])
开发者ID:gimlids,项目名称:LTPM,代码行数:2,代码来源:conf_test.py


示例5: test_empty

 def test_empty(self):
     assert_equal(combine_lists(), [])
开发者ID:gimlids,项目名称:LTPM,代码行数:2,代码来源:conf_test.py


示例6: test_empty

 def test_empty(self):
     self.assertEqual(combine_lists(), [])
开发者ID:icio,项目名称:mrjob,代码行数:2,代码来源:test_conf.py


示例7: test_mix_lists_and_scalars

 def test_mix_lists_and_scalars(self):
     self.assertEqual(combine_lists([1, 2], 3, (4, 5), 6),
                      [1, 2, 3, 4, 5, 6])
开发者ID:Affirm,项目名称:mrjob,代码行数:3,代码来源:test_conf.py


示例8: test_scalars

 def test_scalars(self):
     self.assertEqual(combine_lists(None, False, b'\x00', 42, 3.14),
                      [False, b'\x00', 42, 3.14])
开发者ID:Affirm,项目名称:mrjob,代码行数:3,代码来源:test_conf.py


示例9: test_dicts

 def test_dicts(self):
     self.assertEqual(combine_lists({1: 2}, None, {}),
                      [{1: 2}, {}])
开发者ID:Affirm,项目名称:mrjob,代码行数:3,代码来源:test_conf.py


示例10: test_strings

 def test_strings(self):
     self.assertEqual(combine_lists('one', None, 'two', u'three'),
                      ['one', 'two', u'three'])
开发者ID:Affirm,项目名称:mrjob,代码行数:3,代码来源:test_conf.py


示例11: test_strings

 def test_strings(self):
     self.assertEqual(combine_lists("one", None, "two", u"three"), ["one", "two", u"three"])
开发者ID:kartheek6,项目名称:mrjob,代码行数:2,代码来源:test_conf.py



注:本文中的mrjob.conf.combine_lists函数示例由纯净天空整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。


鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
Python conf.combine_local_envs函数代码示例发布时间:2022-05-27
下一篇:
Python conf.combine_envs函数代码示例发布时间:2022-05-27
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap