- # Licensed to the Apache Software Foundation (ASF) under one or more
- # contributor license agreements. See the NOTICE file distributed with
- # this work for additional information regarding copyright ownership.
- # The ASF licenses this file to You under the Apache License, Version 2.0
- # (the "License"); you may not use this file except in compliance with
- # the License. You may obtain a copy of the License at
- # http://www.apache.org/licenses/LICENSE-2.0
- # Unless required by applicable law or agreed to in writing, software
- # distributed under the License is distributed on an "AS IS" BASIS,
- # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- # See the License for the specific language governing permissions and
- # limitations under the License.
- # The default url filter.
- # Better for whole-internet crawling.
- # Each non-comment, non-blank line contains a regular expression
- # prefixed by '+' or '-'. The first matching pattern in the file
- # determines whether a URL is included or ignored. If no pattern
- # matches, the URL is ignored.
- # skip file: ftp: and mailto: urls
- # skip image and other suffixes we can't yet parse
- # for a more extensive coverage use the urlfilter-suffix plugin
- # skip URLs containing certain characters as probable queries, etc.
- # skip URLs with slash-delimited segment that repeats 3+ times, to break loops
- # accept anything else
nutch 1.7 regex-urlfilter.txt
a guest Nov 1st, 2013 36 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
RAW Paste Data