The crawler encountered some problems

After the code optimization of golang Climbing Zhenai.com, the following error was reported after running. It took half an hour to find the reason. Record it here.

image

The code is like this:< /p>

There is an interface type Parser:

type Parser interface {Parser(contents []byte, url string) ParserResult Serialize() (funcName string, args interface{ })}

There is a FuncParser of type struct:

type FuncParser struct {parser ParserFunc funcName string}

FuncParser implements the Parser interface :

func (f *FuncParser) Parser(contents []byte, url string) ParserResult {return f.Parser(contents, url)}func (f *FuncParser) Serialize() (funcName string, args interface{}) {return f.funcName, nil}

Aside from the overall complexity of the crawler code, simplify the code to the following:

type ParserFunc func(url string) stringtype FuncParser struct {parser ParserFunc}func (f *FuncParser) Parser(url string) string {return f.Parser(url)}func main() {funcParse := FuncParser{ func(url string) string {return url },} func Parse.Parser("http://www.zhenai.com/zhenghun")}

The same error will be reported after running the code:

runtime: goroutine stack exceeds 1000000000-byte limit 
fatal error: stack overflow

runtime stack:
runtime.throw(0x467297, 0xe)
D:/Program Files/Go/go103/src/runtime/ panic.go:616 +0x88
runtime.newstack()
D:/Program Files/Go/go103/src/runtime/stack.go:1054 +0x72d
runtime.morestack()
D:/Program Files/Go/go103/src/runtime/asm_amd64.s:480 +0x91

This example is very obvious. A recursive call is formed in FuncParser’s Parser. Self),
recursively calling itself causes stack overflow, resulting in an error. It should be changed to this: (lowercase parser)

image.png< /p>

In fact, Recursive Call has been prompted in goland

image.png

If you accidentally write this code, look at the following code:

package mainimport ("fmt")type Str stringfunc ( s Str) String() string {return fmt.Sprintf("Str: %s", s)}func main() {var s Str = "hi" fmt.Println(s)}

The same Error:
image.png

You are implementing Str.String in terms of itself. return fmt.Sprintf(“Str: %s”, s) will call s.String(), resulting in infinite recursion. Convert s to string first.

This is working as intended, you are using the %s verb to call Str’s String method, which uses fmt.Sprint to call Str’s String method, and so on.

As follows:

image.png

In fact, there will be warnings about this issue in goland:
image.png

It seems that when you write code, you should pay attention to warnings.

For the project code, please see: https://github.com/ll837448792/crawler



This official account is freeprovided csdn download service, massive IT learning resources, If you are ready to enter the IT pit and inspire to become an excellent programmer, then these resources are suitable for you, including but not limited to java, go, python, springcloud, elk, embedded , Big data, interview materials, front-end and other resources. At the same time, we have formed a technical exchange group. There are a lot of bigwigs in it, and they will share technical articles from time to time. If you want to learn and improve together, you can reply [2] in the background of the official account, and invite and add technical exchanges for free The group learns and improves from each other, and will share programming IT-related resources from time to time.


Scan the QR code to follow, and the exciting content will be pushed to you as soon as possible

image

WordPress database error: [Table 'yf99682.wp_s6mz6tyggq_comments' doesn't exist]
SELECT SQL_CALC_FOUND_ROWS wp_s6mz6tyggq_comments.comment_ID FROM wp_s6mz6tyggq_comments WHERE ( comment_approved = '1' ) AND comment_post_ID = 2057 ORDER BY wp_s6mz6tyggq_comments.comment_date_gmt ASC, wp_s6mz6tyggq_comments.comment_ID ASC

Leave a Comment

Your email address will not be published.